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The objective of the current effort is to provide a computational framework for design and 
analysis of the entire fuel supply system of a liquid rocket engine, including high-fidelity 
unsteady turbopump flow analysis. This capability is needed to support the design of pump 
sub-systems for advanced space transportation vehicles that are likely to involve liquid 
propulsion systems. To date, computational tools for design/analysis of turbopump flows are 
based on relatively lower fidelity methods. An unsteady, three-dimensional viscous flow 
analysis tool involving stationary and rotational components for the entire turbopump 
assembly has not been available for real-world engineering applications. The present effort 
provides developers with information such as transient flow phenomena at start up, and non- 
uniform inflows, and will eventually impact on system vibration and structures. In the proposed 
paper, the progress toward the capability of complete simulation of the turbo-pump for a liquid 
rocket engine is reported. The Space Shuttle Main Engine (SSME) turbo-pump is used as a test 
case for evaluation of the hybrid MPI/Open-MP and MLP versions of the INS3D code. CAD to 
solution auto-scripting capability is being developed for tubopump applications. The relative 
motion of the grid systems for the rotor-stator interaction was obtained using overset grid 
techniques. Unsteady computations for the SSME turbo-pump, which contains 114 zones with 
34.5 million grid points, are carried out on Origin 3000 systems at NASA Ames Research 
Center. Results from these time-accurate simulations with moving boundary capability are 
presented along with the performance of parallel versions of the code. 

INTRODUCTION ^ . 

Until recently, the high performance pump design process was not significantly different from that of 30 years ago. During the 
past 30 years a vast amount of experimental and operational experience has demonstrated that there are many important 
features of pump flows, which are not accounted for in the semi-empirical design process. Pumps being designed today are no 
more technologically advanced than those designed for the Space Shuttle Main Engine (SSME). During that same time span 
huge strides have been made in computers, numerical algorithms, and physical modeling. 

Rocket pumps involve full and partial blades, tip leakage, and exit boundary to diffuser. In addition to the geometric 
complexities, a variety of flow phenomena are encountered in turbopump flows. These include turbulent boundary layer 
separation wakes, transition, tip vortex resolution, three-dimensional effects, and Reynolds numbers effects. In order to 
increase the role of Computational Fluid Dynamics (CFD) in the design process, the CFD analysis tools must be evaluated and 
validated so that designers gain confidence in their use. As a part of extending the CFD technology to pump design work, 
INS3D code has been validated for pump component analysis using data from a rocket pump inducer, and pump impellers . 

When rotating and stationary parts are included as shown in Figure 1, unsteady interactions between stationary and rotating 
components need to be included. To resolve the complex geometry in relative motion, an overset grid approach is employed in 
the current effort. In this approach, a geometrically complex body is decomposed into a number of simple grid components. In 
order to solve the entire configuration including inlet guide vanes, impeller blades, and diffuser blades, an overset grid system 


was constructed using 34.3 Million grid points with 1 14 zones. Since neighboring grids may overlap arbitrarily, these grids can 
be created independently from each other. Thus it is easier to generate high quality and nearly orthogonal grids using this 
approach. Connectivity between neighboring grids is established by interpolation at the grid outer boundaries. Addition of new 
components to the system and simulation of arbitrary relative motion between multiple bodies are achieved by establishing new 
connectivity without disturbing the existing grids. 



Figure 1 . Geometry of a SSME pump including inlet guide vanes, impeller and diffuser blades. 

NUMERICAL METHOD 

The present computations are performed utilizing the INS3D computer code, which solves the incompressible Navier- 
Stokes equations for both steady-state and unsteady flows. The numerical solution of the incompressible Navier-Stokes 
equations requires special attention in order to satisfy the divergence-free constraint on the velocity field because the 
incompressible formulation does not yield the pressure field explicitly from the equation of state or through the continuity 
equation. One way to avoid the numerical difficulty originated by the elliptic nature of the problem is to use an artificial 
compressibility method, developed by Chorin 3 . The artificial compressibility algorithm, which introduces a time-derivative 
of the pressure term into the continuity equation; the elliptic-parabolic type partial differential equations are transformed 
into the hyperbolic-parabolic type. An incompressible flow solver, INS3D, has been developed 4 based on this algorithm. 
Since the convective terms of the resulting equations are hyperbolic, upwind differencing can be applied to these terms. The 
current version uses Roe’s flux-difference splitting 5 . The third and fifth-order upwind differencing used here is an 
implementation of a class of high-accuracy flux-differencing schemes for the compressible flow equations. To obtain time- 
accurate solutions, the equations are iterated to convergence in pseudo-time for each physical time step until the divergence 
of the velocity field has been reduced below a specified tolerance value. The total number of sub-iteration required varies 
depending on the problem, time step size and the artificial compressibility parameter used, and typically ranges from 10 to 
30 sub-iterations. The matrix equation is solved iteratively by using a non-factored Gauss-Seidel type line-relaxation 
scheme 6 , which maintains stability and allows a large pseudo-time step to be taken. Details of the numerical method can be 
found in [4]. The GMRES scheme has also been utilized for the solution of the resulting matrix equation . Computer 
memory requirement for the flow solver INS3D with line-relaxation is 35 times the number of grid points in words, and 
with GMRES-ILU(O) scheme is 220 times the number of grid points in words. When a fast converging scheme, such as a 
GMRES-ILU(O) solver, was implemented into the artificial compressibility method, previous computations showed that 
reasonable agreement was obtained between computed results and experimental data . 

TURBOPUMP SCRIPTS 

For each component of a turbopump (inlet guide vanes, impeller and diffuser), a script was developed to rapidly perform 
the various steps of the simulation process prior to the flow solver call. The starting point is the geometry definition of a 
component which consists of the hub and shroud profile curves and the surface panels of each distinct blade. Functions 
provided by the script include overset surface and volume grid generation, creation of object X-ray maps for hole-cutting 
[9], and creation of overset domain connectivity inputs. After running the domain connectivity program OVERFLOW- 
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D/DCF [10] with the script-created input file, the resulting grids and iblank connectivity information are used as input or 
the INS3D flow solver. Figure 2 illustrates the procedures automatically handled by the scripts. Important attributes o 
each component are parameterized in the scripts. For example, parameterized geometric attributes include the number o 
blades for the inlet guide vanes and the diffuser; and the number of distinct blades per section and the number of sections 
for the impeller. Parameterized grid attributes include hub and shroud surface grid spacings, leading and trailing edge 
spacings, viscous wall spacing for the volume grids, stretching ratios and marching distances for the hyperbolic surface and 
volume grids. All parameters are controlled via an input file and the script can be rapidly re-run to generate di erent 
configurations. For example, the user can input different geometry definitions for the hub, shroud and blades, try different 
number of blades and grid resolutions. 


INLET GUIDE VANES (N repeated blades, no tip clearance) 











Inlet Guide Vanes 

Manual 

Script (coarse) 

Script (fine) 

No. of points (millions) 

7.1 

5.8 

1.1 

User time 

1 day 

43 sec. 

20 sec. 


Impeller 

Manual 

Script (coarse) 

Script (fine) 

No. of points (millions) 

19.2 

15.2 

8.8 

User time 

2 weeks 

319 sec. 

234 sec. 

i 

Diffuser 

Manual 

Script (coarse) 

Script (fine) 

No. of points (millions) 

8.0 

6.4 

1.6 

User time 

1 day 

37 sec. 

22 sec. 


Table 1. Comparison of process time from geoemtry definition to domain connectivity inputs. User time for 
the manual process is the total wall clock time for working through the different parts of the process. User 
time for the scripts is the wall clock time to run the scripts on an R12000 300MHz Silicon Graphics 
workstation. 

PARALLEL COMPUTATIONS 

Parallel computing strategies vary depending on computer architecture such as memory arrangement relative to processing 
units. Two approaches have been implemented in the present study: the first approach is hybrid MPI/OpenMP and the 
second one is Multi Level Parallelism (MLP) developed at NAS A- Ames Research Center. The first approach is obtained by 
using massage-passing interface (MPI) for inter-zone parallelism , and by using OpenMP directives for intra-zone 
parallelism 13 . ENS3D-MPI 12 is based on the explicit massage-passing interface across MPI groups and is designed for 
coarse grain parallelism. The primary strategy is to distribute the zones across a set of processors. During the iteration, all 
the processors would exchange boundary condition data between processors whose zones shared interfaces with zones on 
other processors. A simple master-worker architecture was selected because it is relatively simple to implement and it is a 
common architecture for parallel CFD applications. All I/O was performed by master MPI process and data was distributed 
to the workers. After the initialization phase is complete, the program begins its main iteration loop. Figure 3 shows the 
MPI/OpenMP scalability for SSME impeller computations using 19.2 Million grid points. 



Number of CPUs 

Figure 3. INS3D-MPI/OpenMP performance versus CPU counts on SGI Origin platform. 

The second approach in Multi-Level Parallelism (MLP) is obtained by using NAS-MLP 14 routines. The shared memory 
MLP technique developed at NASA Ames Research Center has been incorporated into the INS3D code. This approach 
differs from the MPI/OpenMP approach in a fundamental way in that it does not use messaging at all. All data 
communication at the coarsest and finest level is accomplished via direct memory referencing instructions. This approach 
also provides a simpler mechanism for converting legacy code, such as INS3D, then MPI. For shared memory MLP, the 
coarsest level parallelism is supplied by spawning of independent processes via the standard UNIX fork. The advantage of 
the UNIX fork over MPI procedure is that the user does not have to change the initialization section of the large production 
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code. Library of routines are used to initiate forks, to establish shared memory arenas, and to provide synchronization 
primitives. The shared memory organization for INS3D is shown in figure 4. The boundary data for the overset grid system 
is archived in the shared memory arena by each process. Other processes access the data from the arena as needed. Figure 
5 shows the MLP scalability for SSME impeller computations using 19.2 Million grid points. 


MLP Process 1 MLP Process 2 

Common /focal/ aa,bb Common /local/ aa,bb 



Common /global/ x,y,z 


Figure 4. Shared memory MLP organization for INS3D-MLP. 



Figure 5. Time (sec) per iteration for. SSME impeller computations using 
ENS3D-MLP on SGI Origin platforms. 


Using INS3D-MLP parallel implementation, time-accurate computations for SSME-rigl configuration have been carried 
out on SGI Origin 2000 and 3000 platforms. Instantaneous snapshots of particle traces and pressure surfaces from these 
computations are shown in Figure 6. This procedure provides flow-field details not readily available from experiments. The 
initial conditions for these simulations were the flow was at rest and the impeller started to rotate impulsively. Three full 
impeller rotations were completed in the simulations using 34.3 Million grid points. Using 128 SGI Origin 3000 CPU s, 
one impeller rotation was competed in less then 3.5 days. When 480 processors were used, one impeller rotation was 
completed in less than 1.5 days. One impeller rotation could be completed in 42 days in 1999. More than 30 times speed- 
up has been obtained over 1999 c apabilities. 


5 







Figure 6. Snapshots of particle traces and pressure surfaces from unsteady turbopump computations. 


SUMMARY 

An incompressible flow solver in steady and time-accurate formulations has been utilized for parallel turbo-pump 
computations. Numerical procedures and turbopump scripts for rapid grid system generation and for overset connectivity input 
generation are outlined. Performance of the INS3D-MPI/OpenMP and INS3D-MLP versions of the flow solver for SSME 
impeller model are presented. Using INS3D-MLP parallel implementation, time-accurate computations for SSME-rigl model, 
which includes 1 14 zones with 34.5 Million grid points, have been carried out on SGI Origin platforms. More than 30 times 
speed-up has been obtained over 1999 capability for these computations. 
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Introduction 
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• Turbopump 

- Liquid rocket engine is likely to be used for the next generation 
reusable launch vehicle 

- Efficient, reliable and cost-effective turbopump is a crucial 
element of the new propulsion system 

i.e. $10 ,000/lb => $1,000/ lb payload 
Crew loss down to 1/10,000 mission 

• Current Design Procedure 

- Existing method is semi-empirical, and no more technologically 
advanced than those for the Space Shuttle Main Engine (SSME) 

- During the past 30 years a vast amount of experimental and 
operational has been accumulated 
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Turbopump for Liquid Propulsion System* 
Space Shuttle Main Engine / Advanced Engine 


TURBOPUMP IN SSME 
POWERHEAD 


MAIN ENGINE 


% 




Impeller 


Diffuser 






Scope of Present Work 




• Computcnonal Approach 

- There are many important flow features not fully accounted for 
in earlier design processes 

- During the same period, huge strides have been made in 
computers, numerical methods/codes, and physical modeling 

• Flow Simulation Challenges 

- Geometry is complex, including 

full and partial blades, tip clearance, high-speed impeller 
and stationary diffuser 

- Variety of flow phenomena are encountered such as 
boundary layer separation, transition, tip vortex, wake and 
cavitation 


Scope of Present Work 


• Existing CFD Tools 

- There exist CFD tools for design and analysis primarily based on 
rotational -steady formulation 

• Scope of Current Work 

Current effort is focused on developing an unsteady, 3-D 
viscous flow analysis tool 

- Stationary and rotational components are included for 
simulating the entire turbopump assembly 

- Simulation time from a given geometry to solution is the main 
metrics 

- SSME turbopump is used for test 




Computational Challenges 
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• Validation 

- Experimental data for time-dependant flow are rare 

• Computational challenges 

- Time -integration scheme, convergence 

- Moving grid system, zonal connectivity 

• Computer science 

As the computing resources are changed to parallel and distributed 
platforms, computer science aspects become important, such as 

- Parallel coding and scalability (algorithmic & implementation) 

- Portability, transparent coding etc. 

• Computing resources 

- "Grid" computing will provide new computing resources for 
problem solving environment 

- High-fidelity flow analysis is likely to be performed using "super 
node" which is largely based on parallel architecture 


Flow Solver - INS3D 
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• Time- integration scheme 

Artificial Compressibility Formulation 

- Introduce a pseudo-time level and artificial compressibility 

- Iterate the equations in pseudo-time for each time step until 
incompressibility condition is satisf led. 

Pressure Projection Method 

- Solve auxiliary velocity field f irst, 

- then enforce incompressibility condition by solving a Poisson 
ecuation for pressure. 











Tim® (a®c) par Iteration 





Time (sec) per iteration 
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Parallel Implementation of INS3D 



INS3D-MLP (MAS MLP no pin-to-node) 
/ OpenMP 


TEST CASE : SSME Impeller 
60 zones / 19.2 Million points 




Number ol CPUs 




High-fidelity Simulation of 2 nd Sen RLV Turbopump 



I nlet Snide Vtms 


23 Zones 
6.5 M Points 


► Mcjor Technical Issues 

Pump codes exist, mostly in rotational 
frame of reference, for quick design 
analysis 

Fully 3-D, transient capability is needed 
to advance pump technology 

To make a timely impact on turbopump 
systems development, wall-clock time 
from CAD to solution has to be short 
enough for design evaluation 


Impeller 

long blades / 
medium blades / 
short blades 

60 Zones / 

19.2 M Grid Points 


31 Zones 
8 6 M Poii 



CFD Need 

fcapid grid generation 

Accelerated solution time 
(parallel implementation) 

Large data set management in multiple 
sites (transmission and storage) 

F eature extraction tool 
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Shuttle Upgrade SSME-rigl 






mz 




wm 


background grid 

Impeller Grid 

60 Zones / 19.2 Million Grid Points 

Smallest zone ; 75K /Largest zone : 99 6K 

Less than 156 orphan points. 


grid for tip 
clearance 


Hub grid 
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Scripting for Acceleration of Grid Generation 



Motivation 

Significant user’s effort needed 
in complex process from geometry * v 
to flow solver impeller, \ 


Objective 

Develop script system to 

- generate grids 

- create domain connectivity input 

- create flow solver Input dmusBr ^ 

for different components a,n 



automatically fc E0METW fl — » j Turbopump Scripts! -— (QC f)-H fLOW SOLVER ) 


Approach 


I grids I {domain connoctlvtty Input] [flow aotver Input) 


Develop one script for each component with ring Interface 
between components => easy plug in for different designs 
and combinations of components 


A 

Amts to 


iam toiurdt Zmtot 


Scripting for Acceleration of Grid Generation 




INLET GUIDE VANES AND DIFFUSER 

Old IGV New IGV Old DIFF New DIFF 
No. of points (million) 7.1 1.1 8.0 1.6 


Time to build 1/2 day 10 sec. 1/2 day 8 sec. 

Script tFinlngs on new grids based on SGI R12k 300MHz processor 
Time to build script = 1 day for IGV, 1 day for DIFF 
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SSME-rigl / Initial start 



After 2 1/2 rotation: 



Particle Traces and Pressure 


• Status 

- 34.3 Million Points 

- 400 physical time steps in one rotation. 

- One physical time-step requires less then 
12 minutes wall time with 128 CPU's on 
Origin platforms. One complete rotation 
requires 3.5-days wall- clock time with 
128 processors dedicated to the task. 

- I/O and memory management are critical 
for wall -clock time reduction 

• Issues / Needs 

- In reality, more than 10% of the 
supercomputing facility to one task is not 
always practical. 

- Need lOOx bigger supernode or use lower- 
fidelity method 

- Commun cation to/from designers and 
experimental group is a part of critical 
tecnnol ogies (in Grid computing) 


A 
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Parallel Implementation of INS3D 



RLV 2 nd Gen Turbo pump 

INS3D-MLP / 40 Groups i !4 Zones / 343 M grid points 


Per processor Mflop is 
between 60-70. 

Code optimization for 
cache based platforms 
is currently underway. 
Target Mflops is to 
reach 120 per 
processor. 

Increasing number of 
OpenMP threads is also 
the main objective for 
this effort. 


40 100 200 300 

Number ol CPUs 





RLV Turbopump 
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34.3 Million Grid points RLV Turbopump one impeller rotation 




1 1 1 1 1 j T 1 1 T 1 j 

42 days 

MPl coarse grain 
*^32 CPUs O2K(250MHh) 

T 1— 

, , . 

' - 

' \ 

MPlOpenMP Hybrid 
^ 128 CPU s 02 K(400Mh 






MLP/OpenMI 
-♦ 428 CPUs 02 

K.(400MHz) 

— 

— 


Vi 

L28 CPUs C 

>3K 400Mhz) 





MLP/c! 
L28 CPUs 

jiche optimized^ 
b3K(400MHz) \ 

- 

i i i l 1 • 

U:, ; J. 'W.A 

I 13 day* 

M Lip/Cache opKirnized V 
480 C?Us 03K(400MHz| # 

i i -i : 1 1 * ! L_ 
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Test Case 1 : Consortium Impeller-Diffuser 
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-Geometry and operating conditions obtained on January 24, 2002 

-Impeller : long blades and short blades, + Diffuser 
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Test Case 1 : Consortium Impeller-Diffuser 



Second Rotation : t/T =1.2 

pressure surfaces total velocity surfaces 


PRESSURE 


VELOCITY MAGNITUDE 




A Summary 

Arm Research Carter — 

• Three-dimensional unsteady flow simulation procedure has been developed 
for turbopump in liquid propulsion system 

- MP I/OpenMP hybrid parallelism, and MLF shared memory parallelism has 
bee n implemented in incompressible flow solver, INS3D 

- Scripting capability from CAD geometry to solution has been developed 
anc evaluated for two different conf igurations 

Unsteady flow computations for RLV 2 nd Gen baseline turbopump are 

performed by using 34.3 Million grid points model 

=> more than 30 times speed-up has been obtained over 1999 baseline 

• Future work 

Unsteady validation and design applications are in progress 






